Skip to content

feat: add normalized chemistry results materialized view and update related configurations#584

Merged
jirhiker merged 9 commits into
stagingfrom
ogc-chemistry
Mar 7, 2026
Merged

feat: add normalized chemistry results materialized view and update related configurations#584
jirhiker merged 9 commits into
stagingfrom
ogc-chemistry

Conversation

@jirhiker

@jirhiker jirhiker commented Mar 4, 2026

Copy link
Copy Markdown
Member

Summary

This PR adds normalized chemistry materialized views for water wells, exposes them as pygeoapi collections, hardens startup/CI behavior, and aligns docs/config for Docker/ENV usage.

Primary changes

1. Major chemistry normalized view (water wells)

Updated existing migration:

What it now does:

  • Sources major chemistry only
  • Restricts rows to thing_type = 'water well'
  • One row per current location/thing
  • Uses most recent result per analyte (analysis_date fallback to collection_date)
  • Static analyte columns (not JSON)
  • Matching *_units static columns
  • Single date column: latest_chemistry_date
  • Expanded analyte alias mapping to match current DB codes (Ca, Ca(total), SO4, IONBAL, TAn, TCat, Na+K, etc.)

2. New minor chemistry normalized view (water wells)

Added migration:

Creates:

  • ogc_minor_chemistry_wells materialized view with:
    • water-well-only filter
    • one row per current location/thing
    • latest-per-analyte selection
    • static analyte value columns
    • matching *_units columns
    • latest_chemistry_date
    • analyte_count
    • unique index for concurrent-safe refresh patterns

pygeoapi collection changes

Updated:

Changes:

  • Renamed collection title:
    • normalized_chemistry_results -> Major Chemistry (Water Wells)
  • Added new collection:
    • minor_chemistry_wells -> Minor Chemistry (Water Wells)

CLI refresh defaults

Updated:

Added to default refresh set:

  • ogc_normalized_chemistry_results
  • ogc_minor_chemistry_wells

Test expectation updates:

OGC test updates

Updated collection assertions/endpoints:

Startup/Compose/CI improvements

Fail-fast entrypoint

Updated:

  • entrypoint.sh
  • Added set -eu so migration failures stop startup before Uvicorn launches.

Compose service/profile adjustments

Updated:

Changes:

  • db_test is now opt-in (profiles: ["test"])
  • Kept hardcoded DB names:
    • dev: ocotilloapi_dev
    • test: ocotilloapi_test

CI service name fix

Updated:

Changes:

  • docker compose build db -> docker compose build db_dev
  • docker compose up -d db -> docker compose up -d db_dev

Documentation/env alignment (Option B)

Updated:

Changes:

  • .env.example now aligns with Compose dev DB name (POSTGRES_DB=ocotilloapi_dev)
  • README now documents:
    • required/optional env vars
    • Compose service names (db_dev, db_test profile)
    • hardcoded dev/test DB names

Validation highlights

  • Verified major view now includes only water wells and populates non-TDS analytes after alias mapping fixes.
  • Verified minor view creation and population in current DB; no non-water-well rows.
  • CLI refresh test updates are in place; local targeted run currently blocked by missing local ocotilloapi_test DB in this environment.

Copilot AI review requested due to automatic review settings March 4, 2026 17:55

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 884ffc66eb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docker-compose.yml Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new pygeoapi-exposed OGC collection backed by a PostgreSQL materialized view that pivots “major chemistry” results into normalized/static analyte columns, and wires it into refresh tooling and local dev config.

Changes:

  • Add Alembic migration creating ogc_normalized_chemistry_results materialized view + unique index for refresh support.
  • Register the new collection in core/pygeoapi-config.yml and include it in the CLI refresh list.
  • Update tests and local Docker/dev startup configuration to account for the new view/collection.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/test_ogc.py Ensures the new normalized_chemistry_results collection appears and its items endpoint works.
tests/test_cli_commands.py Updates refresh-matview command expectations to include the new matview and updated count.
entrypoint.sh Makes DB readiness check configurable via env, and updates log message.
docker-compose.yml Splits DB into db_dev and db_test, updates app wiring to db_dev, and renames volumes.
core/pygeoapi-config.yml Registers normalized_chemistry_results OGC resource pointing at ogc_normalized_chemistry_results.
cli/cli.py Adds ogc_normalized_chemistry_results to default refresh list.
alembic/versions/b6f7a8b9c0d1_add_normalized_chemistry_results_materialized_view.py New migration creating the normalized chemistry results materialized view + index.

You can also share your feedback on Copilot code review. Take the survey.

Comment thread entrypoint.sh
Comment thread docker-compose.yml Outdated
Comment thread docker-compose.yml
Copilot AI review requested due to automatic review settings March 4, 2026 20:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.


You can also share your feedback on Copilot code review. Take the survey.

Comment thread entrypoint.sh
Comment thread README.md Outdated
Comment thread README.md
Copilot AI review requested due to automatic review settings March 4, 2026 21:01
@jirhiker jirhiker removed the request for review from Copilot March 4, 2026 21:01
Copilot AI review requested due to automatic review settings March 4, 2026 21:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.


You can also share your feedback on Copilot code review. Take the survey.

Comment thread alembic/versions/c7f8a9b0d1e2_add_minor_chemistry_wells_materialized_view.py Outdated
Comment thread tests/test_ogc.py Outdated
Comment thread docker-compose.yml
Copilot AI review requested due to automatic review settings March 4, 2026 21:39
@jirhiker jirhiker removed the request for review from Copilot March 4, 2026 21:39
@jirhiker jirhiker merged commit 54a6686 into staging Mar 7, 2026
3 checks passed
@jirhiker jirhiker deleted the ogc-chemistry branch March 12, 2026 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants